Crate datafusion_functions

source ·
Expand description

Function packages for DataFusion.

This crate contains a collection of various function packages for DataFusion, implemented using the extension API. Users may wish to control which functions are available to control the binary size of their application as well as use dialect specific implementations of functions (e.g. Spark vs Postgres)

Each package is implemented as a separate module, activated by a feature flag.

§Available Packages

See the list of modules in this crate for available packages.

§Using A Package

You can register all functions in all packages using the register_all function.

To access and use only the functions in a certain package, use the functions() method in each module.

// get the encoding functions
use datafusion_functions::encoding;
for udf in encoding::functions() {
  registry.register_udf(udf)?;
}

Each package also exports an expr_fn submodule to help create Exprs that invoke functions using a fluent style. For example:

// create an Expr that will invoke the encode function
use datafusion_expr::{col, lit};
use datafusion_functions::expr_fn;
// Equivalent to "encode(my_data, 'hex')" in SQL:
let expr = expr_fn::encode(col("my_data"), lit("hex"));

§Implementing A New Package

To add a new package to this crate, you should follow the model of existing packages. The high level steps are:

  1. Create a new module with the appropriate ScalarUDF implementations.

  2. Use the macros in macros to create standard entry points.

  3. Add a new feature to Cargo.toml, with any optional dependencies

  4. Use the make_package! macro to expose the module when the feature is enabled.

Modules§

  • core
    Core datafusion expressions Enabled via feature flag core_expressions “core” DataFusion functions
  • crypto
    “crypto” DataFusion functions
  • datetime
    Date and time expressions. Contains functions such as to_timestamp Enabled via feature flag datetime_expressions date & time DataFusion functions
  • encoding
    Encoding expressions. Contains Hex and binary encode and decode functions. Enabled via feature flag encoding_expressions
  • expr_fn
    Fluent-style API for creating Exprs
  • macros
  • math
    Mathematical functions. Enabled via feature flag math_expressions “math” DataFusion functions
  • regex
    Regular expression functions. Enabled via feature flag regex_expressions “regx” DataFusion functions
  • string
    “string” DataFusion functions
  • unicode
    “unicode” DataFusion functions

Functions§